← All Tags
# databases → natural language queries → sql → spider benchmark → text to sql

1 link tagged with all of: databases + natural language queries + sql + spider benchmark + text to sql

Links

If You Think You Can Do Real-World Text-to-SQL – Communications of the ACM

The article examines why translating natural language into SQL for live data warehouses remains a tough problem. It traces the rise of ad-hoc queries in 1990s‐era historical data stores and shows how schema complexity and performance needs outpace current text-to-SQL systems.
  • Benchmarks like Spider (max 200 tables, single database per task) don't reflect real enterprise warehouses with hundreds of tables and thousands of columns spanning dozens of schemas
  • Production metadata is messy: cryptic/inconsistent column naming, typos, abbreviations, and mixed-language query logs undermine model performance
  • Vendor-specific SQL extensions (Oracle's LISTAGG, PostgreSQL's JSONB operators) and user-defined types routinely break academic text-to-SQL models
  • The authors call for shared benchmarks built from anonymized real industry schemas/workloads and argue LLMs need integration with metadata services and human feedback loops before replacing analysts in critical settings
Last saved Jul 24, 2026 · 1 min read
text to sql spider benchmark natural language queries sql databases